Information processing device and information processing method

ABSTRACT

An information processing device includes an acquisition unit that acquires manipulation information based on a gesture manipulation of a user and a determination unit that determines that processing is to be performed on a basis of the manipulation information. The determination unit determines that first processing is to be performed in a case where first manipulation information based on a first user manipulation is acquired, and that the first processing is to be performed, on a basis of acquired usage information of the user, in a case where second manipulation information based on a second user manipulation related to the first user manipulation is acquired.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2016/082874 filed on Nov. 4, 2016, which claimspriority benefit of Japanese Patent Application No. JP 2015-233340 filedin the Japan Patent Office on Nov. 30, 2015. Each of theabove-referenced applications is hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present disclosure relates to an information processing device, aninformation processing method, and a program.

BACKGROUND ART

Conventionally, for example, in Patent Literature 1 below, a technologythat assumes performing appropriate gesture feedback using gestureprediction information has been described.

CITATION LIST Patent Literature

Patent Literature 1: JP 2012-8772A

DISCLOSURE OF INVENTION Technical Problem

A motion recognition gesture in a watch-type terminal or other wearableterminals enables manipulation without a user directly manipulating abutton or a touch panel of the terminal, and can provide the user withconvenience. However, in the case where the user's motion is smallerthan assumed, manipulation cannot be carried out in some cases. On theother hand, there is a problem in that relaxing a gesture recognitioncondition increases cases in which a gesture is recognized even in asituation in which a gesture is not intended to be recognized, leadingto a decrease in convenience and satisfaction for the user.

In addition, in the case where the user's motion is similar to anassumed gesture but is not recognized as a gesture, it seems that theuser is highly likely to have desired gesture recognition, but underpresent circumstances, the only way to cause gesture recognition is totry the same gesture again.

In addition, the user uses the terminal in various situations, and thereis a problem in that uniformly applying a gesture recognition conditionmakes it difficult to perform gesture recognition in accordance with asituation, leading to a decrease in the user's convenience.

Hence, there have been demands that a user's convenience be enhanced bygesture recognition with high precision in accordance with a user'susage situation.

Solution to Problem

According to the present disclosure, there is provided an informationprocessing device including: an acquisition unit configured to acquiremanipulation information based on a gesture manipulation of a user; anda determination unit configured to determine that processing is to beperformed on a basis of the manipulation information. The determinationunit determines that first processing is to be performed in a case wherefirst manipulation information based on a first user manipulation isacquired, and that the first processing is to be performed, on a basisof acquired usage information of the user, in a case where secondmanipulation information based on a second user manipulation related tothe first user manipulation is acquired.

In addition, according to the present disclosure, there is provided aninformation processing method including: acquiring manipulationinformation based on a gesture manipulation of a user; and determiningthat processing is to be performed on a basis of the manipulationinformation. It is determined that first processing is to be performedin a case where first manipulation information based on a first usermanipulation is acquired, and that the first processing is to beperformed, on a basis of acquired usage information of the user, in acase where second manipulation information based on a second usermanipulation related to the first user manipulation is acquired.

In addition, according to the present disclosure, there is provided aprogram causing a computer to function as: a means of acquiringmanipulation information based on a gesture manipulation of a user; anda means of determining that processing is to be performed on a basis ofthe manipulation information. The program causes the computer tofunction as a means of determining that first processing is to beperformed in a case where first manipulation information based on afirst user manipulation is acquired, and that the first processing is tobe performed, on a basis of acquired usage information of the user, in acase where second manipulation information based on a second usermanipulation related to the first user manipulation is acquired.

Advantageous Effects of Invention

According to the present disclosure, gestures can be recognized withhigh precision without impairment of a user's convenience.

Note that the effects described above are not necessarily limitative.With or in the place of the above effects, there may be achieved any oneof the effects described in this specification or other effects that maybe grasped from this specification.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram illustrating an appearance of a terminaldevice according to an embodiment of the present disclosure.

FIG. 2 is a block diagram illustrating a functional configuration of aterminal device.

FIG. 3A is a schematic diagram illustrating an example in which adisplay screen of a display unit 102 is lit when an arm is swung up.

FIG. 3B is a schematic diagram illustrating an example in which adisplay screen of a display unit is lit when an arm is swung up.

FIG. 3C is a schematic diagram illustrating an example in which adisplay screen of a display unit is lit when an arm is swung up.

FIG. 3D is a schematic diagram illustrating an example in which adisplay screen of a display unit is lit when an arm is swung up.

FIG. 4 is a block diagram illustrating a functional configurationimplemented by a microcomputer for gesture triggering.

FIG. 5 is a flowchart illustrating processing for implementing theprocessing illustrated in FIGS. 3A, 3B, 3C, and 3D.

FIG. 6 is a characteristic diagram illustrating the relationship betweena detection value of an acceleration sensor and a feature value v.

FIG. 7 is a flowchart illustrating an example in which in the case wherean action that a user has performed for gesture triggering (swinging upan arm) does not satisfy a first triggering condition but satisfies asecond triggering condition more relaxed than the first triggeringcondition, information indicating that is displayed on a terminaldevice.

FIG. 8 is a schematic diagram illustrating an example of application toa gesture of tapping a terminal device.

FIG. 9 is a flowchart illustrating an example in which a threshold valueof a triggering condition is changed in accordance with a swing-upaction of a user of a terminal device.

FIG. 10 is a schematic diagram illustrating an example in which thepresent embodiment is applied to a head-mounted display (HMD).

FIG. 11 is a schematic diagram illustrating an example in which, when atelevision receiver (TV) is manipulated, gesture recognition isperformed using a camera provided in the TV or an acceleration sensor ofa remote controller in a hand of a user.

FIG. 12A is a schematic diagram illustrating operation when a messagearrives at a terminal device.

FIG. 12B is a schematic diagram illustrating operation when a messagearrives at a terminal device.

FIG. 12C is a schematic diagram illustrating operation when a messagearrives at a terminal device.

FIG. 12D is a schematic diagram illustrating operation when a messagearrives at a terminal device.

FIG. 13 is a flowchart illustrating processing in a second embodiment.

MODE(S) FOR CARRYING OUT THE INVENTION

Hereinafter, (a) preferred embodiment(s) of the present disclosure willbe described in detail with reference to the appended drawings. Notethat, in this specification and the appended drawings, structuralelements that have substantially the same function and structure aredenoted with the same reference numerals, and repeated explanation ofthese structural elements is omitted.

Note that description will be given in the following order.

1. First Embodiment

1.1. Example of appearance of terminal device

1.2. Functional configuration of terminal device

1.3. Typical example of gesture triggering

1.4. Processing of gesture triggering

1.5. Other examples of processing of gesture triggering

1.6. Examples of application to other devices

2. Second Embodiment

2.1. Changing gesture recognition function according to manipulationhistory of user

2.2. Changing gesture recognition function according to notification

2.3. Changing LP filter according to activity of user

1. First Embodiment

[1.1. Example of Appearance of Terminal Device]

FIG. 1 is a schematic diagram illustrating an appearance of a terminaldevice 100 according to an embodiment of the present disclosure. Thisterminal device 100 is configured as a watch-type device including adisplay unit 102 (display) and a sensor such as an acceleration sensor,and has a website viewing function, an e-mail function, a function oflistening to and watching music, moving images, etc., a positiondetection function using a GPS or the like, a navigation function, andthe like, like a smartphone, a table terminal, etc.

The terminal device 100 constantly senses a value of the sensor such asan acceleration sensor, applies the value of the sensor to a gesturedetermination function (gesture determination condition), and decidestriggering/non-triggering of a gesture according to the result. As themost common example of a gesture, a screen is lit when an arm is swungup. Other examples include gestures such as twisting an arm and tappingan index finger and a middle finger.

[1.2. Functional Configuration of Terminal Device]

FIG. 2 is a block diagram illustrating a functional configuration of theterminal device 100. As illustrated in FIG. 2, the terminal device 100includes, in addition to the display unit (reporting unit) 102, anacceleration sensor 104, a gyro sensor 106, a vibrator (reporting unit)107, a microcomputer 110, a RAM 120, a storage (storage unit) 122, awireless connection unit (wireless communication unit) 124, and a mainCPU 130. The terminal device 100 can connect to a master smartphone ordirectly connect to the Internet, via the wireless connection unit 124.

The acceleration sensor 104 and the gyro sensor 106 are once connectedto the microcomputer 110 for sensor processing, and then connected tothe main CPU 130 (application processor) that performs main processing.The microcomputer 110 for sensor processing may perform simple gesturediscrimination.

While the terminal device 100 is running, the main CPU 130 functions asan application processor, and executes the website viewing function, thee-mail function, the function of listening to and watching music, movingimages, etc., the position detection function using a GPS or the like,the navigation function, and the like, which are mentioned above. On theother hand, in the terminal device 100, in the case where there is nouser manipulation for a certain period of time, power is turned off bytime-up or the like, and the microcomputer 110 constantly sensesdetection values of the acceleration sensor 104 and the gyro sensor 106.Then, the detection values of these sensors are applied to a gesturedetermination function, and triggering/non-triggering of a gesture isdecided according to the result. Since the microcomputer 110 consumessignificantly less power than the main CPU 130, power consumption can besignificantly reduced by stopping the main CPU 130 and keeping only themicrocomputer 110 running when the power is off.

[1.3. Typical example of gesture triggering]

When a gesture is triggered on the basis of a detection value of asensor while the power of the terminal device 100 is off, the power ofthe terminal device 100 is turned on, and a display screen of thedisplay unit 102 is lit. FIGS. 3A, 3B, 3C, and 3D are schematic diagramsillustrating an example in which the display screen of the display unit102 is lit when an arm is swung up, as the most common example of agesture.

Here, as basic operation, the display screen is lit in the case where adetection value of the acceleration sensor 104 when an arm is swung upis larger than a determination value of a gesture for lighting a screen.On the other hand, in the case where the user tries to light the displayscreen but the detection value of the sensor does not reach thedetermination value, having the user perform the same action againrequires an action troublesome for the user and leads to a decrease inconvenience. Therefore, in the present embodiment, when a detectionvalue of a sensor when an arm is swung up is slightly smaller than adetermination value of a gesture for lighting a screen, the screen islit when the user moves a wrist slightly and then stops the movement.This operation is illustrated in FIGS. 3A, 3B, 3C, and 3D.

First, as illustrated in FIG. 3A, the user swings up the terminal device100 to try triggering of a gesture, but the terminal device 100 is notswung up enough, and the screen is not lit as illustrated in FIG. 3B,and the gesture is not triggered. In this case, by the user performingan additional small gesture of bringing the terminal device 100 closerto the user as illustrated in FIG. 3C, the display screen of the displayunit 102 is lit as illustrated in FIG. 3D, and the user can achieve adesired operation of lighting of the display screen by the gesture.

In FIGS. 3A, 3B, 3C, and 3D, first, second, and third determinationconditions (gesture triggering conditions) are described. The firstdetermination condition, the second determination condition, and thethird determination condition allow, respectively, invocation(triggering) by swinging up of an arm, swinging up of an arm on acondition more relaxed than the first determination condition, andchanging an angle of an arm or lightly shaking the arm in a state wherethe arm is swung up. In addition, in a finger tap manipulation, thefirst determination condition is a manipulation of tapping an indexfinger and a middle finger, the second determination condition isdetection of a tap manipulation on a condition more relaxed than thefirst determination condition, and the third determination condition istap detection on a condition that is stricter than the seconddetermination condition but more relaxed than the first determinationcondition. Note that a tap manipulation will be described later.

According to processing illustrated in FIGS. 3A, 3B, 3C, and 3D, in thecase where the user tries to light the display screen by a gesture, andthe gesture is not triggered because an arm swing-up amount is small,the gesture can be triggered by performing an additional action. Thus,in the case where a gesture is not triggered, the user can trigger thegesture with a small action without swinging up the arm again.

On the other hand, when a determination value of a gesture for lightinga screen is made small, the gesture can be triggered even in the casewhere an arm swing-up amount is small. However, in this case, thegesture is assumed to be triggered even in the case where the user doesnot intend to trigger the gesture, such as when the user holds a strapin a train, for example.

Hence, a technique of the present embodiment can reliably suppressunintended gesture triggering and, in the case where a gesture is nottriggered, reliably trigger the gesture with an additional action.

[1.4. Processing of Gesture Triggering]

FIG. 4 is a block diagram illustrating a functional configurationimplemented by the microcomputer 110 for gesture triggering. Asillustrated in FIG. 4, as a configuration for gesture triggering, theacceleration sensor 104, a sensor monitoring module 230, a gesturedetermination module 240, and a display module (processing unit) 220 areincluded.

In the example illustrated in FIG. 4, a sensor value of the accelerationsensor 104 is processed by the sensor monitoring module 230. The sensormonitoring module 230 performs only relatively simple processing, andcan operate with low power consumption. In addition, the gesturedetermination module 240 is started up and caused to perform processingas necessary. In the case where a motion is detected by the sensormonitoring module 230, the gesture determination module 240 performsgesture determination, and in the case where the motion is recognized asa gesture, the processing unit 220 performs notification to the displaymodule and various applications.

Processing of the sensor monitoring module 230 can be divided into amotion noise removal unit 214, a determination function acquisition unit216, and a threshold value determination unit 218. First,three-dimensional input from the acceleration sensor 104 is subjected tonoise removal by the motion noise removal unit 214, and is converted toa one-dimensional value such as norm calculation by the determinationfunction acquisition unit 216. Furthermore, comparison with a thresholdvalue is performed by the threshold value determination unit 218, and inthe case where a threshold value condition is satisfied, the gesturedetermination module 240 is started up to perform detailed processing.An example of the motion noise removal unit 214 is a low-pass filter(LPF), and removes a high-frequency component (motion noise) such asnoise from a detection value of the acceleration sensor 104. Structuralelements illustrated in FIG. 4 can be configured using a circuit(hardware), or the microcomputer 110 and a program (software) forcausing it to function. The motion noise removal unit 214 includes alow-pass filter (LPF), for example.

FIG. 5 is a flowchart illustrating processing for implementing theprocessing illustrated in FIGS. 3A, 3B, 3C, and 3D. The processingillustrated in FIG. 5 is mainly performed in the microcomputer 110.First, in step S10, the user performs a first manipulation. Here, thefirst manipulation corresponds to an action of swinging up an arm. Next,in step S11, a detection value of the acceleration sensor 104 isacquired. In next step S12, a feature value v to be used for firstgesture determination is calculated. In next step S14, it is determinedwhether the feature value v satisfies a gesture triggering condition,and in the case where the gesture triggering condition is satisfied, theprocessing advances to step S16 and a gesture is triggered.

FIG. 6 is a characteristic diagram illustrating the relationship betweena detection value of the acceleration sensor 104 and the feature valuev. In FIG. 6, characteristics indicated by two types of broken linesindicate sensor values of the acceleration sensor 104, and indicate anacceleration xy in an xy-axis direction and an acceleration z in az-axis direction. A sensor value acquisition unit 212 acquires thecharacteristics indicated by the broken lines in FIG. 6 in step S11 ofFIG. 5. Characteristics indicated by two types of solid lines (a thickline and a thin line) in FIG. 6 indicate sensor values of theacceleration sensor 104 after noise removal by the motion noise removalunit 214; the thin line indicates a sensor value LP(xy) in the xy-axisdirection, and the thick line indicates a sensor value LP(z) in thez-axis direction. The determination function acquisition unit 216acquires the characteristics indicated by the solid lines in FIG. 6,which have been subjected to removal of a high-frequency component suchas noise by the motion noise removal unit 214 and converted toone-dimensional values, in step S12 of FIG. 5. The characteristicsindicated by the solid lines in FIG. 6 correspond to the feature valuev.

In addition, in FIG. 6, characteristics indicated by two types ofdot-dash lines indicate threshold values for determining thecharacteristics indicated by the solid lines, and indicate a thresholdvalue TH_xy for determining characteristics in the xy-axis direction anda threshold value TH_z for determining characteristics in the z-axisdirection. The threshold value determination unit 218 compares thecharacteristics LP(xy), LP(z) indicated by the solid lines in FIG. 6with the threshold values TH_xy, TH_z, and determines whether thegesture triggering condition is satisfied on the basis of whether thevalues of the characteristics LP(xy), LP(z) exceed the threshold valuesTH_xy, TH_z in step S14 of FIG. 5. It is possible to determine whetherthe first manipulation (the manipulation of swinging up the terminaldevice 100) satisfies the gesture triggering condition by setting thethreshold values TH_xy, TH_z to desired values in advance andregistering them. When it is determined that the gesture triggeringcondition is satisfied by the threshold value determination unit 218,the processing unit 220 performs gesture triggering processing. Inaddition, when the gesture triggering condition is satisfied, theprocessing unit 220 performs processing of changing the state of anapplication and outputting information indicating that the triggeringcondition has been satisfied.

In the case where the feature value v does not satisfy the gesturetriggering condition in step S14 of FIG. 5, the processing advances tostep S18. In step S18, it is determined whether the feature value vsatisfies a second triggering condition, and in the case where thesecond triggering condition is satisfied, the processing advances tostep S20. The second triggering condition is a condition for detecting agesture similar to the first triggering condition, and is a conditionmore relaxed than the first triggering condition. Alternatively, thesecond triggering condition may be a condition for detecting a gesturedifferent from the first triggering condition. On the other hand, in thecase where the second triggering condition is not satisfied in step S18,the processing is ended. In step S18, the threshold value determinationunit 218 performs determination using threshold values of a more relaxeddetermination condition than the threshold values TH_xy, TH_z.

In next step S20, when the user performs a second manipulation, adetection value of the acceleration sensor 104 is acquired in next stepS21. Here, the second manipulation corresponds to a gesture of bringingthe terminal device 100 closer to the user as illustrated in FIG. 3C.After step S21, the processing advances to step S22, and a feature valueu to be used for second gesture determination is calculated. Like thefeature value v, the feature value u corresponds to the characteristicsLP(xy), LP(z) indicated by the solid lines in FIG. 6.

In next step S24, it is determined whether the feature value u satisfiesa third triggering condition, and in the case where the third triggeringcondition is satisfied, the processing advances to step S26 and agesture is triggered. A gesture satisfying the third triggeringcondition is an additional small gesture of bringing the terminal device100 closer to the user as illustrated in FIG. 3C, for example. On theother hand, in the case where the third triggering condition is notsatisfied in step S24, the processing is ended.

In step S24, the threshold value determination unit 218 performsdetermination using threshold values different from the threshold valuesTH_xy, TH_z. The gesture of bringing the terminal device 100 closer tothe user as illustrated in FIG. 3C can be detected by setting athreshold value TH_xy for determining characteristics in the xy-axisdirection and a threshold value TH_z for determining characteristics inthe z-axis direction to desired values in advance and registering them.

In the processing of FIG. 5, gesture determination for the firstmanipulation is performed on the basis of the feature value v, and inthe case where the feature value v does not satisfy the first normaltriggering condition (step S14), the second triggering condition ischecked (step S18). The second triggering condition is a condition morerelaxed than the first triggering condition, and the second triggeringcondition allows triggering in all cases where the first triggeringcondition allows triggering. In addition, a motion that is similar tothe first triggering condition but does not satisfy the first triggeringcondition also triggers the second triggering condition. In other words,the second triggering condition is a more relaxed condition thatincludes the first triggering condition. In the case where the featurevalue v satisfies the second triggering condition, subsequent values ofthe acceleration sensor 104 are detected (step S21), and in the casewhere the user tries an additional gesture, evaluation is performedusing a third triggering condition different from the first triggeringcondition (step S24), and in the case where this is satisfied, a gestureis triggered (step S26).

Consequently, in the case where an action that the user has performedfor gesture triggering (swinging up an arm) does not satisfy the firsttriggering condition but satisfies the second triggering condition morerelaxed than the first triggering condition, an additional simplegesture is evaluated using the third triggering condition. Then, in thecase where the additional gesture satisfies the third triggeringcondition, a gesture is triggered. Thus, in the case where the usertries to trigger a gesture, and an arm swing-up amount is too small totrigger the gesture, the gesture can be reliably triggered by performingan additional gesture. Note that in the case where an action that theuser has performed for gesture triggering (swinging up an arm) satisfiesthe second triggering condition, a gesture is triggered in the casewhere an additional gesture is performed at the same time as the actionthat the user has performed for gesture triggering (swinging up an arm)or within a predetermined period of time from the action that the userhas performed for gesture triggering. In this case, there is assumed tobe a time lag between the execution of the action that the user hasperformed for gesture triggering (swinging up an arm) and the actualacquisition of its action information, or a time lag between theexecution of the additional gesture and the actual acquisition of itsaction information. Therefore, a case where manipulation information ofswinging up of an arm and manipulation information of an additionalgesture are acquired at the same time and a case where they are acquiredat different timings can both be assumed.

[1.5. Other Examples of Processing of Gesture Triggering]

FIG. 7 is a flowchart illustrating an example in which in the case wherean action that the user has performed for gesture triggering (swingingup an arm) does not satisfy the first triggering condition but satisfiesthe second triggering condition more relaxed than the first triggeringcondition, information indicating that is displayed on the terminaldevice 100. In addition, FIG. 7 illustrates an example in which in thecase where an action that the user has performed for gesture triggering(swinging up an arm) does not satisfy the first triggering condition butsatisfies the second triggering condition more relaxed than the firsttriggering condition, the gyro sensor 106 is started up and a triggeringcondition is relaxed.

First, in step S30, the user performs a first manipulation. Here, thefirst manipulation corresponds to an action of swinging up an arm. Next,in step S31, a detection value of the acceleration sensor 104 isacquired. In next step S32, a feature value v to be used for firstgesture determination is calculated. In next step S34, it is determinedwhether the feature value v satisfies a gesture triggering condition,and in the case where the gesture triggering condition is satisfied, theprocessing advances to step S36 and a gesture is triggered.

On the other hand, in the case where the feature value v does notsatisfy the gesture triggering condition in step S34, the processingadvances to step S38. In step S38, it is determined whether the featurevalue v satisfies a second triggering condition, and in the case wherethe second triggering condition is satisfied, the processing advances tostep S40. On the other hand, in the case where the second triggeringcondition is not satisfied in step S38, the processing is ended.

In step S40, the user is notified that a gesture is recognized but athreshold value of gesture triggering is not reached. This notificationto the user can be performed by a method such as display on the displayunit 102, or start-up of the vibrator 107. In other words, the displayunit 102 and the vibrator 107 function as a reporting unit for promptinga second manipulation in next step S43. In addition, the notification tothe user may show the user that the threshold value of gesturetriggering will become smaller, by lighting a backlight of the terminaldevice 100 for a moment or shortly vibrating the vibrator 107.Alternatively, in the case of a finger tap gesture in screen display ora gesture manipulation on television, instead of a swing-up gesture,display prompting the user to perform manipulation again may beperformed on a screen.

In next step S42, a mode of a sensor of the terminal device 100 ischanged, and the gyro sensor 106 is started up. In next step S43, theuser who has received the notification in step S40 performs the secondmanipulation. The second manipulation corresponds to a gesture ofbringing the terminal device 100 closer to the user as illustrated inFIG. 3C.

In next step S44, detection values of the acceleration sensor 104 andthe gyro sensor 106 are acquired.

After step S44, the processing advances to step S46, and a feature valueu to be used for second gesture determination is calculated. In nextstep S47, it is determined whether the feature value u satisfies a thirdtriggering condition, and in the case where the third triggeringcondition is satisfied, the processing advances to step S48 and agesture is triggered. The determination whether the third triggeringcondition is satisfied is performed using a detection value of the gyrosensor 106, in addition to a detection value of the acceleration sensor104. On the other hand, in the case where the third triggering conditionis not satisfied in step S46, the processing is ended. In the case wherethe notification in step S40 has been performed, the user is more likelyto perform the second manipulation; hence, a condition for determiningwhether the third triggering condition is satisfied may be relaxedrelative to the normal condition (step S24 of FIG. 5). Thus, the gesturetriggering condition can be changed dynamically.

According to the processing of FIG. 7, in the case where an action ofthe user (swinging up an arm) does not satisfy the first triggeringcondition but satisfies the second triggering condition more relaxedthan the first triggering condition, notification is performed by amethod such as display on the display unit 102, or driving of thevibrator 107; thus, the user is notified that the triggering conditionwill be relaxed. Consequently, the user can be notified that the firstmanipulation has failed to satisfy the first triggering condition by anarrow margin and that the triggering condition will be relaxed. Theuser who has received the notification can recognize that a gesture istriggered by performing an additional action, and performs the secondmanipulation in step S43. Thus, the gesture can be triggered by thefeature value u attributed to the second manipulation satisfying thethird triggering condition.

In addition, according to the processing of FIG. 7, gesture detection isperformed using only the acceleration sensor 104 in normal operation,but in the case where the first manipulation of the user is similar to aswing-up action, the gyro sensor 106, which is normally not used, canalso be started up to be used for detection of the next secondmanipulation. This can enhance the detection precision of the secondmanipulation, and consequently can relax requirements for gesturedetection. Note that although the triggering condition in detecting thesecond manipulation is relaxed by using the gyro sensor 106 incombination in FIG. 7, it is also possible to relax the threshold valuesTH_xy, TH_z themselves of the determination condition in detecting thesecond manipulation.

In addition, although, in the case where the first manipulation of theuser is similar to a swing-up action, the gyro sensor 106 is alsostarted up to be used for detection of the next second manipulation inthe above-described example, it is also possible to enhance recognitionprecision in detecting the second manipulation by temporarily increasinga sampling frequency of the acceleration sensor 104 or increasing aclock of the microcomputer 110 and the main CPU 130, for example, totemporarily change a mode of the terminal device 100 or the sensor. Theacceleration sensor 104 can obtain a detection value with higherprecision by having its sampling frequency increased, but its powerconsumption is also increased. Therefore, in regard to a newly startedup sensor or a once changed mode, the newly started up sensor is stoppedor the mode is returned to the original mode in the case where certaintime-out time elapses or a gesture is triggered in the second gesturedetection.

Note that although an application example in the case where the userperforms a gesture of swinging up an arm is described in theabove-described example, the present embodiment can also be applied to agesture of detecting that the user has tapped the terminal device 100with a finger, in addition to a gesture of swinging up an arm. FIG. 8 isa schematic diagram illustrating an example of application to a gestureof tapping the terminal device 100, and illustrates an example in afunction of stopping an alarm with a finger tap. In this example, theuser performs a finger tap manipulation of tapping an index finger and amiddle finger to stop the alarm, but the tap is not strong enough andthe operation of stopping the alarm is not actually performed.

For example, in the case where the terminal device 100 causes an alarmto tell a time and the user performs a finger tap manipulation to stopthe alarm, but the tap is too weak to cause a reaction on the firsttriggering condition, a system cannot determine whether this tapmanipulation is an intentional manipulation; hence, if the tap satisfiesthe second triggering condition, it is shown on the screen of thedisplay unit 102 that a tap has been tried, and an animation (Tap Again)prompting another tap is displayed. Furthermore, while the animation isdisplayed, a threshold value for detecting a tap is made lower. Thus,the user can perform the next tap easily, and, in the case where the tapsatisfies the third triggering condition, the next tap can be easilydetected.

On the other hand, even in the case where the tap is too weak to cause areaction on the first triggering condition, when time-out elapses,display of the animation (Tap Again) is extinguished, and the firstdetermination condition is applied again instead of the thirddetermination condition that has been temporarily applied.

As well as displaying a character string as in FIG. 8, it is alsopossible to change a button of the terminal device 100 in color or shapeor display a gesture determination function itself as a gauge tovisually show the user that a slight addition is needed. In addition, inthe case where the present embodiment is applied to a manipulation ofmoving a palm from side to side to turn the page of a menu on a screenon television or a large-screen display, display is performed as if themenu was deformed in the direction in which an arm was moved, therebyprompting the user to make a gesture again. When time-out elapses, themenu is returned from deformation.

[1.6. Examples of Application to Other Devices]

Although a watch-type device is exemplified as the terminal device 100in the above-described example, the present embodiment can also beapplied to a device that is worn on a head and detects a user's motionto manipulate a terminal, such as a head-mounted display or a wirelessreceiver. FIG. 10 is a schematic diagram illustrating an example inwhich the present embodiment is applied to a head-mounted display (HMD)200. In the head-mounted display 200, an internal sensor such as anacceleration sensor, a camera, or an IR can acquire the user's gesturesuch as nodding or shaking one's head. The same applies to the wirelessreceiver. For such a gesture, processing similar to that in thewatch-type device described above can be performed.

FIG. 11 is a schematic diagram illustrating an example in which, when atelevision receiver (TV) 300 (or a projector or the like) ismanipulated, gesture recognition is performed using a camera 310provided in the TV 300 or an acceleration sensor of a remote controllerin a hand of the user. In this example, the user performs a manipulationof moving the body, such as an arm, the head, and so on; thus, thecamera 310 or the acceleration sensor recognizes a gesture, enablingmanipulation of the TV 300. In the case where the camera 310 recognizesthe gesture, in the configuration of FIG. 4, the gesture detected by thecamera 310 is sent to the sensor monitoring module 230 and gesturedetermination is performed. Feedback to the user can be performed usinga screen, sound, or lighting of the TV 300. For example, volume can beraised and lowered by a gesture of raising and lowering a palm, a menucan be selected by moving a palm from side to side, and power can beturned on/off by pointing at the screen. In addition, the presentembodiment can also be applied to gesture manipulation in a game. Forexample, a manipulation of changing a point of view by shaking theremote controller from side to side can be made comfortable by changinga gesture triggering condition in accordance with a scene in the game.As an example of application, the point of view is made difficult tochange during conversation with another player and quickly changeableduring movement. Moreover, a manipulation of attacking with a weapon canbe made easier when an enemy is in front, which enables the game to beplayed comfortably.

2. Second Embodiment

[2.1. Changing Gesture Recognition Function According to ManipulationHistory of User]

Next, a second embodiment of the present disclosure is described. Abasic configuration of the terminal device 100 according to the secondembodiment is similar to that in the first embodiment. In the secondembodiment, a gesture triggering condition is changed in accordance withvarious situations of the terminal device 100. First, changing a gesturerecognition function according to a manipulation history of the user isdescribed.

In the case where the gesture of swinging up an arm described above isperformed, there are individual differences in the strength of theswing-up action among users. In other words, gesture manipulationinvolves an individual difference between, for example, a user whosemotion is clear and a user whose motion is weak. However, it isdifficult to initially customize the terminal device 100 in accordancewith differences between individual users; hence, a gesture reactionfunction is changed to match the user on the basis of manipulation logsduring use. FIG. 9 is a flowchart illustrating processing of changingthe gesture reaction function.

First, in step S50, the user performs a first manipulation. Here, thefirst manipulation corresponds to an action of swinging up an arm. Next,in step S51, a detection value of the acceleration sensor 104 isacquired. In next step S52, a feature value v to be used for firstgesture determination is calculated. In next step S54, it is determinedwhether the feature value v satisfies a gesture triggering condition,and in the case where the gesture triggering condition is satisfied, theprocessing advances to step S56 and a gesture is triggered. Cases wherethe gesture is triggered in step S56 include a case where the conditionof step S54 has been satisfied with a great margin and a case where thecondition has been satisfied with a slight margin. In the former case,the user seems to tend to perform a clear gesture, and in the lattercase, the user seems to tend to perform a gesture with a weak motion.Hence, the processing advances to step S57 after step S56, and thedetermination function used in step S54 is updated on the basis ofinformation indicating whether the condition of step S54 has beensatisfied with a great margin or the condition has been satisfied with aslight margin. This changing is performed every time or each time acertain number of logs are accumulated.

On the other hand, in the case where the feature value v does notsatisfy the gesture triggering condition in step S54, the processingadvances to step S58. In step S58, it is determined whether the featurevalue v satisfies a second triggering condition, and in the case wherethe second triggering condition is satisfied, the processing advances tostep S60. The second triggering condition is a condition more relaxedthan the first triggering condition. On the other hand, in the casewhere the second triggering condition is not satisfied in step S58, theprocessing is ended.

In step S60, since a gesture is not triggered, the user performs amanipulation equivalent to a manipulation that has been tried to beachieved by a gesture, by tapping the screen, manually manipulating apower button to turn on the power of the terminal device 100, etc. Inthe case where this manipulation is performed, it seems that theoriginal gesture manipulation performed in step S50 has been intended bythe user and should have been triggered. Therefore, in step S62, thedetermination function used in step S54 is updated to be relaxed. Innext step S64, the user is notified by screen display that thedetermination function has been updated.

As described above, according to the flowchart of FIG. 9, in the casewhere the first manipulation is not triggered, when the user achieves adesired manipulation by pushing the power button to start up theterminal device 100, for example, a threshold value for determininggesture triggering can be changed from the next time. Consequently, evenin the case where there are individual differences in the strength of anaction among users, the determination condition (threshold value) forgesture triggering can be adjusted optimally. In addition, after thedetermination condition is adjusted as described above, thedetermination condition can be reset to the original value. Resetting ofthe determination condition is performed by the threshold valuedetermination unit 218 of the sensor monitoring module 230 and thegesture determination module 240. For example, resetting can beperformed in the case where there is a user reset manipulation, the casewhere an application is changed, the case where a different meaning isset for the same gesture, the case where an owner is changed, the casewhere information of high airtightness and importance comes to bedisplayed after gesture triggering, and the like.

[2.2. Changing Gesture Recognition Function According to Notification]

Next, changing a gesture recognition function according to notificationis described. As described above, the terminal device 100 has an e-mailfunction. When a message arrives via e-mail, the user is more likely totry to start up the terminal device 100 by a swing-up action. Therefore,in the second embodiment, in gesture triggering by swing-up, a gesturetriggering determination function is changed by using information ofarrival of a message.

Specifically, during a certain period of time from arrival of a messageat the terminal device 100 and notification of this arrival to the userby vibration, a determination condition for an arm swing-up gesture isrelaxed, so that a motion that usually does not cause a reaction isreacted to as swing-up. Thus, in the case where the user swings up anarm to check the message within this certain period of time, the screencan be lit easily and a desired manipulation such as message viewing canbe achieved.

FIGS. 12A, 12B, 12C, and 12D are schematic diagrams illustratingoperation when a message arrives at the terminal device 100. FIG. 12Aillustrates normal display, and a time is displayed on the display unit102 in this state. When a message arrives at the terminal device 100, asillustrated in FIG. 12B, part of the message is displayed on the screenof the display unit 102, together with vibration caused by the vibrator107, to notify the user that the message has arrived. Next, the userswings up an arm or taps the screen, so that the state transitions to astate illustrated in FIG. 12C, and the user can check details of themessage. On the other hand, in the case where the user does not swing upan arm or tap the screen for a certain period of time and the state isleft in a state illustrated in FIG. 12B, the screen transitions to astate of FIG. 12D and returns to clock display, and a simplenotification icon is displayed.

FIG. 13 is a flowchart illustrating processing in the second embodiment.First, in step S70, a message to the user of the terminal device 100arrives. In next step S72, a system of the terminal device 100 notifiesthe user by screen display, LED lighting, vibration, or the like to tellthe user that the message has arrived. In next step S74, the userperforms a swing-up gesture to check the terminal device 100. In nextstep S76, a detection value of the acceleration sensor 104 is acquired,and in next step S77, a feature value v to be used for gesturedetermination is calculated.

In step S78, it is checked whether a certain period of time has elapsedfrom arrival of the last unchecked notification, and in the case wherethere is an unchecked notification within the certain period of time,the processing advances to step S80, and gesture swing-up is determinedusing a second determination function with a determination conditionmore relaxed than a normal condition. As a result, when the swing-up isdetermined to be a gesture, the screen is lit (step S84). At this time,information indicating that a message has arrived is transmitted to themicrocomputer 110 from the wireless communication unit 124, and thedetermination condition is changed by the threshold value determinationunit 218 of the sensor monitoring module 230 and the gesturedetermination module 240 of the microcomputer 110. The processing unit220 performs gesture triggering in accordance with the change in thedetermination condition, and lights the display unit 202. In addition,the processing unit 220 outputs a result of gesture triggering to thedisplay module and various applications. That is, the processing unit220 also functions as an output unit that outputs a result of gesturetriggering. Note that a time at which an unchecked notification has beenacquired is a time related to acquisition of the unchecked notification,and includes both a reception time of the message itself and anotification timing after reception of the message.

On the other hand, in the case where a certain period of time haselapsed in step S78, the processing advances to step S82, and swing-upis determined using a first determination condition, which is normal. Asa result, when the swing-up is determined to be a gesture, the screen islit (step S86).

In this manner, in the case where the user tries to check a messageafter notification of the message, the second determination functionwith a determination condition more relaxed than the normal condition isapplied, which reduces the possibility of the user failing in swing-up,and improves convenience and availability.

In step S78, conditions used for selection of a determination functioninclude the following elements, in addition to message arrival.

(1) Message contents: in the case where the message is addressed to aspecific person, the determination function is relaxed as urgentnotification. In the case of an advertisement mail, the determinationfunction is not relaxed (made stricter).

(2) Incoming call and alarm: in the case where an arm is swung up whilethere is an incoming call, the determination function is relaxed. Thedetermination function is relaxed also when an alarm is ringing.

(3) Personal information of user: for example, the determinationfunction is changed in accordance with an age or a length of an arm.

(4) Environmental information: for example, the determination conditionis changed in accordance with weather.

(5) Schedule, position, and activity: as a result of activity analysisdetermined from a schedule, a GPS, and the acceleration sensor 104, thedetermination function is not relaxed (made stricter) during movieappreciation, for example. The determination function is relaxed duringmovement, between plans, or around the end of a plan. The determinationfunction is not relaxed at midnight. The determination function isrelaxed in the morning. For example, if a time to take a train, anopening time of a meeting, and the like are registered in the terminaldevice 100 in advance, the user is more likely to swing up an arm to seethe display screen on the display unit 102 before that time. Hence,relaxing the gesture determination function enables lighting reliablywhen the user swings up an arm before the time to take the train or themeeting opening time.(6) Application and scene: For the same gesture, the determinationfunction is changed depending on the importance of the manipulation. Forexample, in the case of manipulating a button for sending a message witha finger tap, the determination function is made stricter, because wrongtransmission will bring about a serious problem. Also in the case of amanipulation of deleting a message or a plan, the determination functionis made stricter. On the other hand, for a manipulation of marking anotification as read, the determination function is relaxed, because itis performed frequently and is not of high importance. In addition, inthe case where the application is a game with vigorous motion, thedetermination function is made stricter to make gesture triggeringdifficult. Moreover, the determination function is changed in accordancewith a situation of the game.(7) Activity history and gesture triggering history of user: inaccordance with a position, time, a day of week, etc., the determinationfunction is relaxed for a place, time, and day of week having a historyof the user seeing the screen.(8) Specifications of terminal device: for example, in accordance with aweight of the terminal device 100, the determination function is madestricter for a lighter terminal device because it is easier to swing upan arm.(9) State of battery: in the case where the remaining battery power islow, the display screen can be made difficult to light by making thegesture determination condition stricter, which can suppress a decreasein remaining battery power.

Note that in the display in step S72, display (UI) showing the user thatgesture recognition is made easier may be performed. In addition, aperiod from the determination in step S78 to the advance to step S80 inwhich the second determination function is applied corresponds to theperiod of FIG. 12B, and the user can be aware that a gesture is invokedby simple swing-up in this period.

[2.3. Changing LP Filter According to Activity of User]

Next, changing an LP filter according to an activity of the user isdescribed. In order to perform gesture determination with highprecision, it is necessary to separate a manipulation in which the userhas intentionally performed a gesture from other manipulations. For thisseparation, the motion noise removal unit 214 can be used. Since thelow-pass filter can absorb stationary noise, wrong gesture triggeringduring motion can be prevented, but the application of the filter causesdelay and makes it relatively difficult to cause a reaction immediatelyafter a manipulation. For example, a cutoff frequency of the filter ofthe motion noise removal unit 214 is made lower in the case where theuser is during motion such as walking or running, and the cutofffrequency is made higher or the use of the low-pass filter itself isquit when there is less motion, such as when sitting. In this manner, agesture can be triggered with less latency during stop, while preventingwrong gesture triggering during motion.

The preferred embodiment(s) of the present disclosure has/have beendescribed above with reference to the accompanying drawings, whilst thepresent disclosure is not limited to the above examples. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

Further, the effects described in this specification are merelyillustrative or exemplified effects, and are not limitative. That is,with or in the place of the above effects, the technology according tothe present disclosure may achieve other effects that are clear to thoseskilled in the art from the description of this specification.

Additionally, the present technology may also be configured as below.

(1)

An information processing device including:

an acquisition unit configured to acquire manipulation information basedon a gesture manipulation of a user; and

a determination unit configured to determine that processing is to beperformed on a basis of the manipulation information,

in which the determination unit determines

that first processing is to be performed in a case where firstmanipulation information based on a first user manipulation is acquired,and

that the first processing is to be performed, on a basis of acquiredusage information of the user, in a case where second manipulationinformation based on a second user manipulation related to the firstuser manipulation is acquired.

(2)

The information processing device according to (1), in which the firstuser manipulation and the second user manipulation are similar to eachother.

(3)

The information processing device according to (1) or (2), including

an output unit configured to output a result of determination by thedetermination unit.

(4)

The information processing device according to any one of (1) to (3), inwhich a condition for determining that the first user manipulation hasbeen performed is relaxed on the basis of the usage information of theuser.

(5)

The information processing device according to (4), in which thecondition for determining that the first user manipulation has beenperformed is relaxed within a predetermined period of time from a timerelated to acquisition of the usage information of the user.

(6)

The information processing device according to (4), in which thecondition for determining that the first user manipulation has beenperformed is relaxed within a predetermined period of time from a timerelated to acquisition of received message information.

(7)

The information processing device according to any one of (1) to (6), inwhich the usage information is personal information about the user.

(8)

The information processing device according to any one of (1) to (6), inwhich the usage information is message information received by acommunication unit.

(9)

The information processing device according to any one of (1) to (6), inwhich the usage information includes at least one of a schedule, agesture triggering history, an age, and a length of an arm of the user.

(10)

The information processing device according to any one of (1) to (6), inwhich the usage information is information dependent on a usageenvironment of the user.

(11)

The information processing device according to any one of (1) to (6), inwhich the usage information is information dependent on the own device.

(12)

The information processing device according to any one of (1) to (6), inwhich the usage information is information dependent on an applicationused by the user.

(13)

The information processing device according to any one of (1) to (12),in which the determination unit changes a threshold value fordetermining that the first user manipulation has been performed, inresponse to acquisition of the usage information of the user.

(14)

The information processing device according to claim 13, in which thedetermination unit changes the threshold value in a case where the firstuser manipulation is performed within a predetermined period of timefrom a time related to the acquisition of the usage information.

(15)

The information processing device according to any one of (1) to (14),in which the determination unit dynamically changes a threshold valuefor determining that the first user manipulation has been performed.

(16)

The information processing device according to (15), in which thedetermination unit changes the threshold value each time the first usermanipulation is performed.

(17)

The information processing device according to any one of (1) to (16),in which the gesture manipulation is a non-contact manipulation.

(18)

The information processing device according to any one of (1) to (17),including

a display unit,

in which the first processing is processing of causing the display unitto display information.

(19)

An information processing method including:

acquiring manipulation information based on a gesture manipulation of auser; and determining that processing is to be performed on a basis ofthe manipulation information,

in which it is determined

that first processing is to be performed in a case where firstmanipulation information based on a first user manipulation is acquired,and

that the first processing is to be performed, on a basis of acquiredusage information of the user, in a case where second manipulationinformation based on a second user manipulation related to the firstuser manipulation is acquired.

(20)

A program causing a computer to function as:

a means of acquiring manipulation information based on a gesturemanipulation of a user; and

a means of determining that processing is to be performed on a basis ofthe manipulation information,

in which the program causes the computer to function as a means ofdetermining

that first processing is to be performed in a case where firstmanipulation information based on a first user manipulation is acquired,and

that the first processing is to be performed, on a basis of acquiredusage information of the user, in a case where second manipulationinformation based on a second user manipulation related to the firstuser manipulation is acquired.

REFERENCE SIGNS LIST

-   212 sensor monitoring module-   218 threshold value determination unit-   220 processing unit-   240 gesture determination module

The invention claimed is:
 1. An information processing device,comprising: at least one processor configured to: acquire firstmanipulation information based on a first gesture manipulation of auser; acquire second manipulation information based on a second gesturemanipulation of the user; notify usage information to the user;determine a trigger corresponding to one of the first manipulationinformation or the second manipulation information, wherein the trigger,corresponding to the first manipulation information, is determined whena time period, elapsed from the notified usage information unchecked bythe user, is outside a threshold period of time, the trigger,corresponding to the second manipulation information, is determined whenthe time period, elapsed from the notified usage information uncheckedby the user, is within the threshold period of time, and the secondgesture manipulation is associated with the first gesture manipulation;and execute a process based on the trigger.
 2. The informationprocessing device according to claim 1, wherein the first gesturemanipulation is similar to the second gesture manipulation.
 3. Theinformation processing device according to claim 1, wherein the at leastone processor is further configured to output a result of determinationof the execution of the process.
 4. The information processing deviceaccording to claim 1, wherein the trigger corresponding to the firstgesture manipulation is determined based on a first condition, thetrigger corresponding to the second gesture manipulation is determinedbased on a second condition, and the second condition is relaxed thanthe first condition, based on the usage information.
 5. The informationprocessing device according to claim 4, wherein the second condition forthe determination of the trigger corresponding to the second gesturemanipulation is relaxed within the threshold period of time.
 6. Theinformation processing device according to claim 4, wherein the at leastone processor is further configured to receive message information asthe usage information, and the second condition for the determination ofthe trigger corresponding to the second gesture manipulation is relaxedwithin the threshold period of time.
 7. The information processingdevice according to claim 1, wherein the usage information is personalinformation about the user.
 8. The information processing deviceaccording to claim 1, wherein the at least one processor is furtherconfigured to acquire the usage information as message information. 9.The information processing device according to claim 1, wherein the atleast one processor is further configured to determine the triggercorresponding to the second manipulation information based on at leastone of a schedule, a gesture triggering history, an age, or a length ofan arm of the user.
 10. The information processing device according toclaim 1, wherein the at least one processor is further configured todetermine the trigger corresponding to the second manipulationinformation based on information of a usage environment of the user. 11.The information processing device according to claim 1, wherein the atleast one processor is further configured to determine the triggercorresponding to the second manipulation information based oninformation of a device associated with the user.
 12. The informationprocessing device according to claim 1, wherein the at least oneprocessor is further configured to determine the trigger correspondingto the second manipulation information based on information of anapplication associated with the user.
 13. The information processingdevice according to claim 1, wherein the at least one processor isfurther configured to change a threshold value for the determination ofthe trigger based on the notification of the usage information.
 14. Theinformation processing device according to claim 13, wherein the atleast one processor is further configured to change the threshold valuebased on the acquisition of the second manipulation information withinthe threshold period of time.
 15. The information processing deviceaccording to claim 1, wherein the at least one processor is furtherconfigured to dynamically change a threshold value for the determinationof the trigger.
 16. The information processing device according to claim15, wherein the at least one processor is further configured to changethe threshold value each time of the acquisition of the firstmanipulation information.
 17. The information processing deviceaccording to claim 1, wherein the first gesture manipulation and thesecond gesture manipulation are a non- contact manipulation with respectto the information processing device.
 18. The information processingdevice according to claim 1, further comprising a display device,wherein the execution of the process corresponds to control of thedisplay device to display specific information.
 19. An informationprocessing method, comprising: acquiring first manipulation informationbased on a first gesture manipulation of a user; acquiring secondmanipulation information based on a second gesture manipulation of theuser; notifying usage information to the user; determining a triggercorresponding to one of the first manipulation information or the secondmanipulation information, wherein the trigger, corresponding to thefirst manipulation information, is determined when a time period,elapsed from the notified usage information unchecked by the user, isoutside a threshold period of time, the trigger, corresponding to thesecond manipulation information, is determined when the time period,elapsed from the notified usage information unchecked by the user, iswithin the threshold period of time, and the second gesture manipulationis associated with the first gesture manipulation; and executing aprocess based on the trigger.
 20. A non-transitory computer-readablemedium having stored thereon computer-executable instructions, whichwhen executed by at least one processor of an information processingdevice, cause the at least one processor to execute operations, theoperations comprising: acquiring first manipulation information based ona first gesture manipulation of a user; acquiring second manipulationinformation based on a second gesture manipulation of the user;notifying usage information to the user; determining a triggercorresponding to one of the first manipulation information or the secondmanipulation information, wherein the trigger, corresponding to thefirst manipulation information, is determined when a time period,elapsed from the notified usage information unchecked by the user, isoutside a threshold period of time, the trigger, corresponding to thesecond manipulation information, is determined when the time period,elapsed from the notified usage information unchecked by the user, iswithin the threshold period of time, and the second gesture manipulationis associated with the first gesture manipulation; and executing aprocess based on the trigger.